Interface neural network

ABSTRACT

An operation method of a neural network, a training method, and a signal processing apparatus are provided. The operation method includes receiving an output signal from a first neural network, and converting a first feature included in the output signal to a second feature configured to be input to a second neural network, based on a conversion rule controlling conversion between a feature to be output from the first neural network and a feature to be input to the second neural network. The operation method further includes generating an input signal to be input to the second neural network, based on the second feature, and transmitting the input signal to the second neural network.

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation application of U.S. patent application Ser. No.15/247,160, filed Aug. 25, 2016, in the U.S. Patent and TrademarkOffice, which claims priority from Korean Patent Application No.10-2016-0046426 filed on Apr. 15, 2016, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference in its entirety.

BACKGROUND 1. Field

Methods and apparatuses consistent with example embodiments relate to anoperation method of a neural network and a signal processing apparatususing the neural network.

2. Description of the Related Art

A neural network is used in various fields of technology. Thus, researchis being conducted on technology for a smooth interaction among aplurality of neural networks.

SUMMARY

Example embodiments may address at least the above problems and/ordisadvantages and other disadvantages not described above. Also, theexample embodiments are not required to overcome the disadvantagesdescribed above, and an example embodiment may not overcome any of theproblems described above.

According to an aspect of an example embodiment, there is provided anoperation method of a neural network, the operation method includingreceiving an output signal from a first neural network, and converting afirst feature included in the output signal to a second featureconfigured to be input to a second neural network, based on a conversionrule controlling conversion between a feature to be output from thefirst neural network and a feature to be input to the second neuralnetwork. The operation method further includes generating an inputsignal to be input to the second neural network, based on the secondfeature, and transmitting the input signal to the second neural network.

The first feature may include a first feature vector, and the secondfeature may include a second feature vector.

The neural network may include an interface neural network to which theconversion rule is applied.

An input dimension of the neural network may correspond to an outputdimension of the first neural network, and an output dimension of theneural network may correspond to an input dimension of the second neuralnetwork.

The conversion rule may include parameters of the neural network thatare optimized.

The operation method may further include, in response to the firstneural network being replaced with a third neural network, updating theconversion rule to control conversion between a feature to be outputfrom the third neural network and the feature to be input to the secondneural network, and in response to the second neural network beingreplaced with a fourth neural network, updating the conversion rule tocontrol conversion between the feature to be output from the firstneural network and a feature to be input to the fourth neural network.

The updating the conversion rule to control the conversion between thefeature to be output from the third neural network and the feature to beinput to the second neural network may include adjusting parameters ofthe neural network, based on a relationship between the feature to beoutput from the third neural network and the feature to be input to thesecond neural network, and the updating the conversion rule to controlthe conversion between the feature to be output from the first neuralnetwork and the feature to be input to the fourth neural network mayinclude adjusting parameters of the neural network, based on arelationship between the feature to be output from the first neuralnetwork and the feature to be input to the fourth neural network.

The third neural network and the first neural network may bedistinguished with respect to any one or any combination of an inputmodality, an output modality, an input dimension, an output dimension,an input feature, and an output feature, and the fourth neural networkand the second neural network may be distinguished with respect to anyone or any combination of an input modality, an output modality, aninput dimension, an output dimension, an input feature, and an outputfeature.

In response to the first neural network being replaced with the thirdneural network, a type of an input signal based on the updatedconversion rule may be identical to a type of the input signal based onthe conversion rule, and in response to the second neural network beingreplaced with the fourth neural network, a type of an output signalbased on the updated conversion rule may be identical to a type of theoutput signal based on the conversion rule.

The operation method may further include, in response to a third neuralnetwork being additionally connected to the neural network, generating aconversion rule controlling conversion between a feature to be outputfrom the third neural network and the feature to be input to the secondneural network.

The first neural network may be configured to extract, as the firstfeature, a feature vector from an object, and the second neural networkmay be configured to identify the object, based on the input signal.

The first neural network may be configured to determine, as the firstfeature, a command vector of an actuator, and the second neural networkmay be configured to control the actuator, based on the input signal.

According to an aspect of another example embodiment, there is provideda training method including connecting an input layer of an interfaceneural network to an output layer of a first neural network, connectingan output layer of the interface neural network to an input layer of asecond neural network, and inputting a training sample to an input layerof the first neural network. The training method further includesobtaining an output signal from an output layer of the second neuralnetwork in response to the inputting of the training sample, andtraining the interface neural network, based on the output signal and alabel of the training sample.

An input dimension of the interface neural network may correspond to anoutput dimension of the first neural network, and an output dimension ofthe interface neural network may correspond to an input dimension of thesecond neural network.

The interface neural network may be trained to convert a first featureincluded in an output signal that is output from the first neuralnetwork to a second feature configured to be input to the second neuralnetwork, generate an input signal to be input to the second neuralnetwork, based on the second feature, and transmit the input signal tothe second neural network.

The first neural network may be configured to extract, as the firstfeature, a feature vector from an object, and the second neural networkmay be configured to identify the object, based on the input signal.

The first neural network may be configured to determine, as the firstfeature, a command vector of an actuator, and the second neural networkmay be configured to control the actuator, based on the input signal.

A non-transitory computer-readable medium may store a program includinginstructions to control a processor to perform the method.

According to an aspect of another example embodiment, there is provideda signal processing apparatus including a processor configured toreceive an output signal from a first neural network, and convert afirst feature included in the output signal to a second featureconfigured to be input to a second neural network, based on a conversionrule controlling conversion between a feature to be output from thefirst neural network and a feature to be input to the second neuralnetwork. The processor is further configured to generate an input signalto be input to the second neural network, based on the second feature,and transmit the input signal to the second neural network.

According to an aspect of another example embodiment, there is providedan operation method of a neural network, the operation method includingreceiving an output signal from a first neural network, and converting afirst feature of an object and included in the output signal to a secondfeature of the object and configured to be input to a second neuralnetwork. The operation method further includes generating an inputsignal to be input to the second neural network, the input signalincluding the second feature of the object, and transmitting the inputsignal to the second neural network.

The operation method further includes converting a first command of anactuator and included in the output signal to a second command of theactuator and configured to be input to the second neural network, andgenerating the input signal to be input to the second neural network,the input signal including the second command of the actuator.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects will be more apparent and more readilyappreciated from the following detailed description of exampleembodiments, with reference to the accompanying drawings of which:

FIG. 1 is a flowchart illustrating an operation method of a neuralnetwork, according to an example embodiment;

FIG. 2 is a conceptual diagram illustrating an interface neural networkaccording to an example embodiment;

FIG. 3 is a diagram illustrating an interface neural network accordingto an example embodiment;

FIGS. 4A and 4B are conceptual diagrams illustrating an interface neuralnetwork to which a new neural network is connected, according to exampleembodiments;

FIG. 5 is a diagram illustrating replacement of a neural network,according to an example embodiment;

FIG. 6 is a diagram illustrating replacement of a neural network,according to another example embodiment;

FIG. 7A is a flowchart illustrating a training method according to anexample embodiment;

FIG. 7B is a diagram illustrating a training method according to anexample embodiment; and

FIG. 8 is a diagram illustrating a signal processing apparatus accordingto an example embodiment.

DETAILED DESCRIPTION

Example embodiments are described in greater detail below with referenceto the accompanying drawings.

In the following description, like drawing reference numerals are usedfor like elements, even in different drawings. The matters defined inthe description, such as detailed construction and elements, areprovided to assist in a comprehensive understanding of the exampleembodiments. However, it is apparent that the example embodiments can bepracticed without those specifically defined matters. Also, well-knownfunctions or constructions may not be described in detail because theywould obscure the description with unnecessary detail.

The terminology used herein is for the purpose of describing the exampleembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms“include/comprise” and/or “have,” when used in this disclosure, specifythe presence of stated features, integers, steps, operations, elements,components, or combinations thereof, but do not preclude the presence oraddition of one or more other features, integers, steps, operations,elements, components, and/or groups thereof. In addition, the terms suchas “unit,” “-er (-or),” and “module” described in the specificationrefer to an element for performing at least one function or operation,and may be implemented in hardware, software, or the combination ofhardware and software.

Terms such as first, second, A, B, (a), (b), and the like may be usedherein to describe components. Each of these terminologies is not usedto define an essence, order or sequence of a corresponding component butused to distinguish the corresponding component from other component(s).For example, a first component may be referred to a second component,and similarly the second component may also be referred to as the firstcomponent.

If it is described in the specification that one component is“connected,” “coupled,” or “joined” to another component, a thirdcomponent may be “connected,” “coupled,” and “joined” between the firstand second components, although the first component may be directlyconnected, coupled or joined to the second component.

Unless otherwise defined, all terms, including technical and scientificterms, used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure pertains. Terms,such as those defined in commonly used dictionaries, are to beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art, and are not to be interpreted in anidealized or overly formal sense unless expressly so defined herein.

Example embodiments to be described hereinafter may be embodied invarious forms of products, for example, a personal computer (PC), alaptop computer, a tablet PC, a smartphone, a television (TV), a smarthome appliance, an intelligent vehicle, a kiosk, and a wearable device.For example, the example embodiments may be applicable to userrecognition in, for example, a smartphone, a mobile device, and a smarthome system. In addition, the example embodiments may be applicable to apayment service based on user recognition. Further, the exampleembodiments may also be applicable to a smart vehicle system that isautomatically started through user recognition.

Example embodiments to be described hereinafter may use an interfaceneural network. The interface neural network refers to a neural networkthat connects, for example, two or more systems, modules, and networks.The systems, modules, and networks may operate independently from oneanother. The systems, modules, and networks may perform functions at thesame levels, or perform functions at a relatively higher or lower level.The systems, modules, and networks may operate based on respectivetraining-based parameters. The example embodiments may providetechnology for training only the interface neural network withouttraining an entire system, although the two or more systems, modules,and networks are connected.

The interface neural network may be trained based on a preset trainingset, or trained based on a trial and error based experience. Theinterface neural network may include a feedforward net. For example, theinterface neural network may include a fully-connected net in a singlelayer or a multilayer. Alternatively, the interface neural network mayinclude a stateful network. For example, the interface neural networkmay include a recurrent neural network.

Hereinafter, for convenience of description, scenarios that theinterface neural network connects a first neural network and a secondneural network will be described. However, modifications may be made tosuch scenarios so that the interface neural network connects, forexample, various systems, modules, and networks. For example, theinterface neural network may connect an existing system and a newmodule. The existing system may operate based on a parameter trained tobe suitable for a user, and the new module may operate based on aparameter trained to be suitable for a function of the module. Accordingto example embodiments, even in such a case that the new module isconnected to the existing system, training only the interface neuralnetwork may be needed without training an entire system again.

FIG. 1 is a flowchart illustrating an operation method of a neuralnetwork, according to an example embodiment.

An operation method of a neural network to be described with referenceto FIG. 1 may be performed by a signal processing apparatus. The signalprocessing apparatus may be embodied by a software module, a hardwaremodule, or a combination thereof.

According to an example embodiment, the signal processing apparatus mayreceive a signal output from a first neural network, generate a newsignal by processing the received signal, and transmit the generatedsignal to a second neural network. Here, the first neural network mayreceive a first input signal, and generate a first output signal byprocessing the received signal. The second neural network may receive asecond input signal, and generate a second output signal by processingthe received signal.

Referring to FIG. 1, in operation 101, the signal processing apparatusreceives the first output signal output from the first neural network.In operation 102, the signal processing apparatus converts a featureincluded in the first output signal to a feature suitable for an inputof the second neural network based on a conversion rule that controlsconversion between a first feature to be output from the first neuralnetwork and a second feature to be input to the second neural network.In operation 103, the signal processing apparatus generates the secondinput signal corresponding to the second neural network based on thefeature obtained from the conversion.

The conversion rule between the first feature to be output from thefirst neural network and the second feature to be input to the secondneural network refers to a rule for changing the feature included in thefirst output signal and processing the feature to be a signal that maybe processed by the second neural network. Using such a predefinedconversion rule, the signal processing apparatus may provide aninterface that enables compatibility between the first neural networkand the second neural network that handle different types of signals.

The signal processing apparatus may load the conversion rule from aninternal or external memory or a server, perform the conversion on thefeature of the first output signal, and generate the second input signalsuitable for the input of the second neural network. Alternatively, thesignal processing apparatus may receive, from the internal or externalmemory or the server, information associated with an output of the firstneural network and information associated with the input of the secondneural network, and generate the conversion rule between the two neuralnetworks. Here, the conversion rule may not be limited to a rule definedbetween the output of the first neural network and the input of thesecond neural network, but various modifications may be made. Forexample, the conversion rule may include parameters of the interfaceneural network.

The conversion rule may be defined or generated by any one or anycombination of parameters of the interface neural network, theparameters including a structure of an input or output layer of at leastone neural network connected to the signal processing apparatus, a typeof input or output data, a dimension of an input or output vector, and amodality of an input or output. However, information used to define orgenerate the conversion rule for compatibility among a plurality ofneural networks may not be limited to the foregoing.

According to an example embodiment, the signal processing apparatus mayadapt the first feature to be output from the first neural network tothe second feature to be input to the second neural network. The signalprocessing apparatus may change the feature included in the first outputsignal output from the first neural network and generate the secondinput signal to be input to the second neural network. Here, the firstoutput signal may be a feature vector to be output from the first neuralnetwork, and the second input signal may be a feature vector to be inputto the second neural network.

For example, the second neural network may be an upper neural networkthat is trained to perform an operation. In such an example, the signalprocessing apparatus may change the feature included in the first outputsignal of the first neural network, which is a lower neural network, tobe suitable for the operation to be performed by the second neuralnetwork, using the interface neural network. The second neural networkmay perform the operation based on the second input signal generatedthrough the conversion of the feature included in the first outputsignal.

The signal processing apparatus may convert the feature included in thefirst output signal to the feature of a form that may be processed bythe second neural network, and generate the second input signal. Togenerate the second input signal, the signal processing apparatus mayuse the interface neural network to convert the feature of the firstoutput signal to the feature corresponding to the input of the secondneural network, and the interface neural network may apply theconversion rule for the conversion. For the conversion of the featureincluded in the first output signal, elements, for example, a type ofthe feature corresponding to the second neural network, an order inwhich features are input, and a location at which the feature is input,may be applied. The signal processing apparatus may convert the featureincluded in the first output signal to the feature corresponding to theinput of the second neural network, using the conversion rule to whichsuch elements are applied, and the second neural network may operatebased on the feature obtained through the conversion. For example, thesecond neural network may recognize a pattern of the feature obtainedthrough the conversion and generate a result of the recognition.

In operation 104, the signal processing apparatus transmits the secondinput signal to the second neural network. The first output signaloutput from the first neural network may not be a signal that issuitable for being input data of the second neural network, and thus thesignal processing apparatus may perform the conversion on the featureincluded in the first output signal to generate the second input signal.The signal processing apparatus may transmit, to the second neuralnetwork, the second input signal generated corresponding to the input ofthe second neural network. For example, the second input signal maymatch an input layer of the second neural network, and be generatedbased on a type of a signal that is processed by the second neuralnetwork.

FIG. 2 is a conceptual diagram illustrating an interface neural networkaccording to an example embodiment.

Neural networks may be applicable to various technical fields ineveryday life. Numerous types of devices and software applications thatoperate based on neural network technology, and robots embodied by acombination thereof, may also be applicable to various fields. Accordingto an example embodiment, an apparatus including a plurality of neuralnetworks may provide functions through interactions performed by theneural networks being connected to one another.

The neural networks may be hierarchically embodied. For example, a lowerneural network may extract a feature using information obtained from asensor at an end of the apparatus including the neural networks, and anupper neural network may identify an object based on the extractedfeature. Alternatively, the upper neural network may generate adetermination result based on input information and generate a controlcommand based on the generated determination result, and the lowerneural network may operate in response to the control command receivedfrom the upper neural network.

According to an example embodiment, each of the neural networks mayprocess an input signal based on a designed intention, and generate anoutput signal. Examples of a neural network may include a sensoryfeature extractor, a modality specific encoder and decoder, a memory,and a motor controller.

In an entire network system into which the neural networks areintegrated, a neural network may be replaced with another neural networkor added. For example, one among the neural networks may be replacedwith a neural network of which performance is improved or a neuralnetwork of which a modality corresponds to a different sensor. Also, aneural network that may process a new type of an input or output signalthat may be added.

A plurality of neural networks integrated with a sensor or motor havingan improved performance may be provided in an off-the-shelf form byvarious vendors. For example, when one among neural networks included inan entire network system is a convolutional neural network (CNN)integrated with a vision sensor to detect a feature, a new type ofsensor may be integrated with the CNN, or the CNN may be replaced with aneural network having the same sensor with an improved performance.

When a new neural network is integrated into the entire network system,the entire network system integrated with the new neural network mayneed to be trained, while a state of the previous entire network systemis being maintained. However, if the entire network system integratedwith the new neural network needs to be trained overall, retraining eachneural network may be needed for connection among the neural networksincluded in the entire network system, and thus a loss of previouslytrained information may occur.

Referring to FIG. 2, an entire network system including a plurality ofneural networks includes a first neural network 201, a second neuralnetwork 202, and an interface neural network 203. The interface neuralnetwork 203 connects the first neural network 201 and the second neuralnetwork 202. The first neural network 201 and the second neural network202 may be independently trained, and the interface neural network 203may be embodied based on information associated with a relationshipbetween the first neural network 201 and the second neural network 202for compatibility between the first neural network 201 and the secondneural network 202.

According to an example embodiment, a conversion rule that controlsconversion between an output of the first neural network 201 and aninput of the second neural network 202 may be applied to the interfaceneural network 203. A first output signal output from the first neuralnetwork 201 may be converted to a second input signal corresponding tothe second neural network 202 through the interface neural network 203to which the conversion rule is applied. The conversion rule refers to arule for processing a received signal by the interface neural network203, and may be defined based on, for example, information associatedwith a protocol between the output of the first neural network 201 andan input of the interface neural network 203 and information associatedwith a protocol between an output of the interface neural network 203and the input of the second neural network 202.

The conversion rule may include information that matches output vectorsdefined by the output of the first neural network 201 to input vectorsdefined by the input of the second neural network 202. Although aprotocol is not defined between the first neural network 201 and thesecond neural network 202, data may be exchanged therebetween via theinterface neural network 203.

For example, when one among the first neural network 201 and the secondneural network 202 connected to the interface neural network 203 isreplaced with a new neural network, the interface neural network 203 mayconnect the new neural network to the existing neural network. Wheneither one or both of the first neural network 201 and the second neuralnetwork 202 is replaced with the new neural network, the signalprocessing apparatus may update the conversion rule of the interfaceneural network 203. To connect the new neural network and the existingneural network, the interface neural network 203 may optimize parametersof the interface neural network 203 and learn the optimized parameters.For example, when the second neural network 202 is replaced with afourth neural network, the signal processing apparatus may update theconversion rule to be a conversion rule that controls conversion betweenthe first feature to be output from the first neural network 201 and afourth feature to be input to the fourth neural network. In this case,the signal processing apparatus may adjust parameters of the interfaceneural network 203 based on a relationship between the first feature andthe fourth feature. Here, the fourth neural network and the secondneural network 202 may be distinguished with respect to any one or anycombination of an input or output modality, an input or outputdimension, and an input or output feature. Further, a type of a firstoutput signal based on the updated conversion rule may be identical to atype of the first output signal based on the previous conversion rule.

In the entire network system into which the new neural network isintegrated, the interface neural network 203 may be solely trained, andremaining neural networks excluding the interface neural network 203 maymaintain previous states.

Although a unilateral data flow between two neural networks connected toan interface neural network is illustrated in FIG. 2, examples providedherein may include bilateral transmission and reception of a signalbetween neural networks connected through an interface neural network,using a method of integrating signals received from a plurality ofneural networks, processing the integrated signals to be a new signal,and transmitting the signal obtained through the processing to anotherneural network. According to an example embodiment, an interface neuralnetwork may connect at least two neural networks for compatibilitytherebetween, various modifications may be made to a form in which theneural networks are connected to the interface neural network, and theform may not be limited to examples described herein.

FIG. 3 is a diagram illustrating an interface neural network accordingto an example embodiment.

Referring to FIG. 3, the interface neural network 203 includes an inputlayer 301 and an output layer 303. According to an example embodiment,the input layer 301 of the interface neural network 203 receives a firstoutput signal from an output layer 302 of the first neural network 201.To receive the first output signal, the input layer 301 of the interfaceneural network 203 matches the output layer 302 of the first neuralnetwork 201. For example, a dimension (e.g., a number of nodes) of theinput layer 301 of the interface neural network 203 may correspond to adimension (e.g., a number of nodes) of the output layer 302 of the firstneural network 201.

The output layer 303 of the interface neural network 203 transmits asecond input signal to an input layer 304 of the second neural network202. To transmit the second input signal, the output layer 303 of theinterface neural network 203 matches the input layer 304 of the secondneural network 202. For example, a dimension (e.g., a number of nodes)of the output layer 303 of the interface neural network 203 maycorrespond to a dimension (e.g., a number of nodes) of the input layer304 of the second neural network 202.

A topology of neurons in the interface neural network 203 may beembodied to easily convert a first output received by the input layer301 to a second input to be transmitted by the output layer 303. Forexample, a conversion rule that controls conversion between an output ofthe first neural network 201 and an input of the second neural network202 may include topology information of the neurons included in theinterface neural network 203, a weighted value of each of the neurons,and information on connections among the neurons, and the interfaceneural network 203 may be defined based on the conversion rule.

According to an example embodiment, when a neural network connected tothe interface neural network 203 is replaced with a new neural networkor a new neural network is additionally connected, either one or both ofthe input layer 301 and the output layer 303 of the interface neuralnetwork 203 may be transformed to match an input or output layer of thenew neural network, and a conversion rule for connecting the existingneural network and the new neural network may be generated or updated.In such a case, the topology information included in the conversion ruledefining the interface neural network 203 may be updated. The interfaceneural network 203 may process a signal between the existing neuralnetwork and the new neural network based on the new conversion rule, andmay connect the existing neural network and the new neural network.

Although an example of presence of an input layer and an output layer ofa neural network is described with reference to FIG. 3, the input layerand the output layer of the neural network may not be separatelypresent, and an interface neural network may be designed to match anyone or any combination of an output and an input of a first neuralnetwork and an output and an input of a second neural network.

FIGS. 4A and 4B are conceptual diagrams illustrating an interface neuralnetwork to which a new neural network is connected, according to exampleembodiments.

Referring to FIG. 4A, when the first neural network 201 connected to theinterface neural network 203 is replaced with a third neural network401, a conversion rule of the interface neural network 203 may beupdated. According to an example embodiment, a signal processingapparatus may update the conversion rule of the interface neural network203 to a new conversion rule that controls conversion between an outputof the third neural network 401 and an input of the second neuralnetwork 202. The signal processing apparatus may adjust parameters ofthe interface neural network 203 based on a relationship between theoutput of the third neural network 401 and the input of the secondneural network 202. The signal processing apparatus may update theprevious conversion rule to the new conversion rule that is optimizedfor the relationship between the output of the third neural network 401and the input of the second neural network 202 based on the adjustedparameters.

A type of a second input signal to be transmitted to the second neuralnetwork 202 from the interface neural network 203 before the firstneural network 201 is replaced with the third neural network 401 may beidentical to a type of a second input signal to be transmitted to thesecond neural network 202 from the interface neural network 201 afterthe first neural network 201 is replaced with the third neural network401. For example, types of features each included in the second inputsignals to be transmitted from the interface neural network 203 beforeand after the replacement may be the same.

Here, the second neural network 202 may be an upper neural network of anentire network system, and the first neural network 201 and the thirdneural network 401 may be a lower neural network in the entire networksystem. In such a case, the interface neural network 203 may provide asignal of the same type, for example, a feature of the same type, to thesecond neural network 202 before and after the replacement. The firstneural network 201 and the third neural network 401 may be differentfrom each other in any one or any combination of an input modality, anoutput modality, an input dimension, an output dimension, a featureincluded in an input signal, and a feature included in an output signal.Although types of features included in signals to be received by theinterface neural network 203 from the lower neural network before andafter the replacement are different, the interface neural network 203may convert a feature included in an output signal of the third neuralnetwork 401 to a feature in a form that may be processed by the secondneural network 202, which is the upper neural network. The interfaceneural network 203 may convert the feature included in the output signalof the third neural network 401 to the feature corresponding to thesecond neural network 202, and transmit, to the second neural network202, the signal of the same type before and after the replacement.

Although the first neural network 201 is replaced with the third neuralnetwork 401 in the entire network system including the first neuralnetwork 201, the second neural network 202, and the interface neuralnetwork 203, the second neural network 202 connected to the interfaceneural network 203 may receive the second input signal of the same typebefore and after the replacement. Further, when the second neuralnetwork 202 is replaced with the fourth neural network, it can beapplied to the exemplary embodiments described above. Thus, onlytraining the interface neural network 203 connected to a new neuralnetwork may be performed without requiring retraining of the neuralnetworks in the entire network system, and thus stability of the entirenetwork system may be ensured.

Referring to FIG. 4B, a third neural network 402 may be additionallyconnected to the interface neural network 203. The signal processingapparatus may generate a conversion rule that controls conversionbetween an input or output of the third neural network 402 and an inputor output of the second neural network 202. In addition, the signalprocessing apparatus may generate a conversion rule that controlsconversion between the input or output of the third neural network 402and an input or output of the first neural network 201. The signalprocessing apparatus may apply the generated conversion rule to theinterface neural network 203.

According to an example embodiment, when the third neural network 402 isadditionally connected to the interface neural network 203, theinterface neural network 203 may receive a signal output from each ofthe first neural network 201 and the third neural network 402, andgenerate a second input signal based on the newly generated conversionrule and each of the received signals. The generated second input signalmay be transmitted to the second neural network 202 from the interfaceneural network 203. For example, in an entire network system, the firstneural network 201 and the third neural network 402 may be a lowerneural network, and may have different modalities from each other. Thesecond neural network 202 may be an upper neural network that may make afinal determination and generate a command using a signal received fromthe interface neural network 203. The first neural network 201 and thethird neural network 402 of the different modalities may extractrespective features from objects, and the interface neural network 203may receive each of the features received from the first neural network201 and the third neural network 402. The interface neural network 203may generate the second input signal based on the respective receivedfeatures and the conversion rule, and the second neural network 202 maymake a determination and generate a command after receiving the secondinput signal from the interface neural network 203. Here, types of thesecond input signals received by the second neural network 202 from theinterface neural network 203 before and after the addition of the thirdneural network 402 may be the same.

FIG. 5 is a diagram illustrating replacement of a neural network,according to an example embodiment.

Referring to FIG. 5, an entire network system includes a neural networkA 505, interface neural networks 508 and 509, a new neural network B506, and a new neural network C 507. Here, the entire network system maybe a robot system. The robot system may include a plurality ofhierarchically connected neural networks, and the interface neuralnetworks 508 and 509 may connect the neural networks.

As illustrated in FIG. 5, an existing vision sensor 501 and an existingactuator 502 are replaced or upgraded with a new vision sensor 503 and anew actuator 504, respectively. The new vision sensor 503 may have animproved performance compared to the existing vision sensor 501.Similarly, the new actuator 504 may have an improved performancecompared to the existing actuator 502. The new vision sensor 503 maygenerate information sensed from an object, and the neural network B 506may generate a visual feature corresponding to the object based on theinformation generated by the new vision sensor 503. Here, the neuralnetwork B 506 may be a CNN that may be trained to generate the visualfeature corresponding to the object, and may extract a favorable featurebased on the improved performance of the new vision sensor 503. The newactuator 504 may be controlled by the neural network C 507. The neuralnetwork C 507 may be a neural network that may be connected to the newactuator 504 and trained to control the new actuator 504.

When two lower neural networks included in the robot system are replacedwith the neural network B 506 and the neural network C 507,respectively, conversion rules of the interface neural networks 508 and509 may be updated. For example, when lower neural networks included inthe robot system are replaced with new neural networks, the interfaceneural networks 508 and 509 may be trained, and remaining neuralnetworks excluding the trained interface neural networks 508 and 509,for example, the neural network A 505, the neural network B 506, and theneural network C 507, may maintain previous states. The robot system mayminimize additional training by adoption of a new neural network usingan interface neural network, for example, the interface neural networks508 and 509.

Here, the interface neural network 508 may generate a conversion rulethat controls conversion between an output of the neural network B 506and an input of the neural network A 505, and update the previousconversion rule to be the generated conversion rule. The interfaceneural network 509 may generate a conversion rule that controlsconversion between an output of the neural network A 505 and an input ofthe neural network C 507, and update the previous conversion rule to bethe generated conversion rule.

The neural network B 506 may output a feature vector corresponding tothe object based on the information sensed by the new vision sensor 503.The interface neural network 508 may convert the feature vector receivedfrom the neural network B 506 to an input signal corresponding to theneural network A 505 based on the updated conversion rule. The interfaceneural network 508 may transmit the input signal obtained through theconversion to the neural network A 505. The neural network A 505 mayidentify or classify the object based on the input signal received fromthe interface neural network 508.

The neural network A 505 may output a command vector of the new actuator504 based on identification information of the object. The interfaceneural network 509 may convert the command vector received from theneural network A 505 to an input signal corresponding to the neuralnetwork C 507 based on the updated conversion rule. The neural network C507 may receive the input signal obtained through the conversion fromthe interface neural network 509, and control the new actuator 504 basedon the received input signal.

FIG. 6 is a diagram illustrating replacement of a neural network,according to another example embodiment.

Referring to FIG. 6, in an entire network system including a neuralnetwork A 601 connected to an interface neural network 602, and a neuralnetwork B 603 connected to the interface neural network 602 and a colorsensor 605, the neural network B 603 is replaced with a neural network C604 connected to a depth sensor 606. The previous entire network system(illustrated in a left portion of FIG. 6) may use the color sensor 605to identify or classify an object. When an environment of identifying anobject changes, the depth sensor 606 may be used in place of the colorsensor 605, and the entire network system adopting the depth sensor 606(illustrated in a right portion of FIG. 6) may use a depth parameter toidentify, classify, detect, or determine an object.

According to an example embodiment, when a sensor connected to a lowerneural network included in an entire network system changes, a modalityof the sensor may change. As described with reference to FIG. 5, whenthe neural network B 603 for a previous modality is replaced with theneural network C 604 for a new modality, the interface neural network602 may update a conversion rule based on an output of the neuralnetwork C 604 and an input of the neural network A 601, and connect theneural network C 604 and the neural network A 601 based on the updatedconversion rule.

FIG. 7A is a flowchart illustrating a training method according to anexample embodiment. FIG. 7B is a diagram illustrating a training methodaccording to an example embodiment.

Referring to FIGS. 7A and 7B, in operation 701, a signal processingapparatus connects an input layer of an interface neural network 730 toan output layer of a first neural network 710. The signal processingapparatus, which is described with reference to FIGS. 1 through 6, mayperform an operation of processing a signal of the interface neuralnetwork 730, and the signal processing apparatus to be described withreference to FIGS. 7A and 7B may train the interface neural network 730.The trained interface neural network 730 may operate as described withreference to FIGS. 1 through 6.

In operation 702, the signal processing apparatus connects an outputlayer of the interface neural network 730 to an input layer of a secondneural network 720. A dimension of the input layer of the interfaceneural network 730 may correspond to a dimension of the output layer ofthe first neural network 710, and a dimension of the output layer of theinterface neural network 730 may correspond to a dimension of the inputlayer of the second neural network 720.

In operation 703, the signal processing apparatus inputs a trainingsample to an input layer of the first neural network 710. For example,the signal processing apparatus may sequentially input a plurality oftraining samples included in a training set to the input layer of thefirst neural network 710. Each of the training samples may include alabel and correspond to a signal to be input to the first neural network710. Label is information to identify training samples. In more detail,supervised learning is the machine learning task of inferring a functionfrom labeled training data. The training data consist of a set oftraining samples. In supervised learning, each sample is a pairconsisting of an input object (typically a vector) and a desired outputvalue (also called the label).

In operation 704, the signal processing apparatus obtains an outputsignal to be output from an output layer of the second neural network720 in response to the input of the training sample. For example, aresult from propagation of the training sample from the first neuralnetwork 710 to the second neural network 720 through the interfaceneural network 730 may be output as the output signal of the secondneural network 720.

In operation 705, the signal processing apparatus trains the interfaceneural network 730 based on the output signal and a label of thetraining sample. The signal processing apparatus may compare a targetoutput signal based on the label of the training sample to the outputsignal output from the second neural network 720, and train theinterface neural network 730 based on a result of the comparison. Thesignal processing apparatus may adjust parameters of the interfaceneural network 730 based on the output signal of the second neuralnetwork 720 and the label. Based on a parameter optimized through theadjusting, a conversion rule that controls conversion between an outputof the first neural network 710 and an input of the second neuralnetwork 720 may be generated, and the interface neural network 730 maybe defined based on the generated conversion rule.

FIG. 8 is a diagram illustrating a signal processing apparatus accordingto an example embodiment.

Referring to FIG. 8, a signal processing apparatus 801 includes a memory802 and a processor 803. The memory 802 may store a program to beexecuted by the processor 803, and temporarily or permanently store datafor processing performed by the processor 803. The processor 803 mayreceive a first output signal output from a first neural network,convert the first output signal to a second input signal correspondingto a second neural network based on a conversion rule that controlsconversion between an output of the first neural network and an input ofthe second neural network, and transmit the second input signal to thesecond neural network. Examples described with reference to FIGS. 1through 7B may be applicable to the signal processing apparatus 801, andthus a more detailed description will be omitted here.

The above-described example embodiments may be recorded innon-transitory computer-readable media including program instructions toimplement various operations that may be performed by a computer. Themedia may also include, alone or in combination with the programinstructions, data files, data structures, and the like. The programinstructions recorded on the media may be those specially designed andconstructed for the purposes of the example embodiments, or they may beof the well-known kind and available to those having skill in thecomputer software arts. Examples of non-transitory computer-readablemedia include magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD ROM discs and DVDs;magneto-optical media such as optical discs; and hardware devices thatare specially configured to store and perform program instructions, suchas read-only memory (ROM), random access memory (RAM), flash memory, andthe like. Examples of program instructions include both machine code,such as code produced by a compiler, and files containing higher levelcode that may be executed by the computer using an interpreter. Thedescribed hardware devices may be configured to act as one or moresoftware modules to perform the operations of the above-describedexample embodiments, or vice versa.

The foregoing example embodiments are examples and are not to beconstrued as limiting. The present teaching can be readily applied toother types of apparatuses. Also, the description of the exampleembodiments is intended to be illustrative, and not to limit the scopeof the claims, and many alternatives, modifications, and variations willbe apparent to those skilled in the art.

What is claimed is:
 1. An operation method of an interface neuralnetwork, the operation method comprising: receiving an output signalfrom a first neural network; generating an input signal to be input to asecond neural network by converting a first feature vector of the outputsignal to a second feature vector of the input signal, based onparameters of the interface neural network controlling conversionbetween a feature vector to be output from the first neural network anda feature vector to be input to the second neural network; andtransmitting the input signal to the second neural network.
 2. Theoperation method of claim 1, wherein an input dimension of the interfaceneural network corresponds to an output dimension of the first neuralnetwork, and an output dimension of the interface neural networkcorresponds to an input dimension of the second neural network.
 3. Theoperation method of claim 1, wherein the parameters of the interfaceneural network are optimized.
 4. The operation method of claim 1,further comprising: based on the first neural network being replacedwith a third neural network, updating the parameters to controlconversion between a feature vector to be output from the third neuralnetwork and the feature vector to be input to the second neural network;and based on the second neural network being replaced with a fourthneural network, updating the parameters to control conversion betweenthe feature vector to be output from the first neural network and afeature vector to be input to the fourth neural network.
 5. Theoperation method of claim 4, wherein the updating the parameters tocontrol the conversion between the feature vector to be output from thethird neural network and the feature vector to be input to the secondneural network comprises adjusting the parameters of the interfaceneural network, based on a relationship between the feature vector to beoutput from the third neural network and the feature vector to be inputto the second neural network, and the updating the parameters to controlthe conversion between the feature vector to be output from the firstneural network and the feature vector to be input to the fourth neuralnetwork comprises adjusting the parameters of the interface neuralnetwork, based on a relationship between the feature vector to be outputfrom the first neural network and the feature vector to be input to thefourth neural network.
 6. The operation method of claim 4, wherein thethird neural network and the first neural network are distinguished withrespect to any one or any combination of an input modality, an outputmodality, an input dimension, an output dimension, an input featurevector, and an output feature vector, and the fourth neural network andthe second neural network are distinguished with respect to any one orany combination of an input modality, an output modality, an inputdimension, an output dimension, an input feature vector, and an outputfeature vector.
 7. The operation method of claim 4, wherein, based onthe first neural network being replaced with the third neural network, atype of an input signal based on the updated parameters is identical toa type of the input signal based on the parameters, and based on thesecond neural network being replaced with the fourth neural network, atype of an output signal based on the updated parameters is identical toa type of the output signal based on the parameters.
 8. The operationmethod of claim 1, further comprising, based on a third neural networkbeing additionally connected to the interface neural network, generatingthe parameters controlling the conversion between the feature vector tobe output from the third neural network and the feature vector to beinput to the second neural network.
 9. The operation method of claim 1,wherein the first neural network is configured to extract the firstfeature vector from an object, and the second neural network isconfigured to identify the object, based on the input signal.
 10. Theoperation method of claim 1, wherein the first neural network isconfigured to determine, as the first feature vector, a command vectorof an actuator, and the second neural network is configured to controlthe actuator, based on the input signal.
 11. An operation method of aninterface neural network, the operation method comprising: receiving afirst signal; generating an input signal to be input to a target neuralnetwork by converting a first feature vector of the first signal to asecond feature vector of the input signal, based on parameters of theinterface neural network controlling conversion between differentfeature vectors; and transmitting the input signal to the target neuralnetwork.
 12. The operation method of claim 11, wherein the parameters ofthe interface neural network are adjusted to control conversion betweena feature vector to be extracted from the first signal and a featurevector to be input to the target neural network based on a relationshipbetween the feature vector to be extracted from the first signal and thefeature vector to be input to the target neural network.
 13. Theoperation method of claim 11, further comprising: based on the firstsignal being replaced with a second signal, updating the parameters tocontrol conversion between a feature vector to be extracted from thesecond signal and the feature vector to be input to the target neuralnetwork; and based on the target neural network being replaced with asecond neural network, updating the parameters to control conversionbetween the feature vector to be extracted from the first signal and afeature vector to be input to the second neural network.
 14. Theoperation method of claim 13, wherein the updating the parameters tocontrol the conversion between the feature vector to be extracted fromthe second signal and the feature vector to be input to the targetneural network comprises adjusting the parameters of the interfaceneural network based on a relationship between the feature vector to beextracted from the second signal and the feature vector to be input tothe target neural network, and the updating the parameters to controlthe conversion between the feature vector to be extracted from the firstsignal and the feature vector to be input to the second neural networkcomprises adjusting the parameters of the interface neural network,based on a relationship between the feature vector to be extracted fromthe first signal and the feature vector to be input to the second neuralnetwork.
 15. The operation method of claim 13, wherein the second signaland the first signal are distinguished with respect to any one or anycombination of a type and a dimension.
 16. The operation method of claim13, wherein, based on the first signal being replaced with the secondsignal, a type of the input signal generated based on the updatedparameters is identical to a type of the input signal generated based onthe parameters.
 17. The operation method of claim 11, wherein an inputdimension of the interface neural network corresponds to a dimension ofthe first signal, and an output dimension of the interface neuralnetwork corresponds to an input dimension of the target neural network.18. A non-transitory computer-readable medium storing a programcomprising instructions to control a processor to perform the method ofclaim
 11. 19. A signal processing apparatus comprising: at least oneprocessor configured to: receive an output signal from a first neuralnetwork; generate an input signal to be input to a second neural networkby converting a first feature vector of the output signal to a secondfeature vector of the input signal, based on parameters of an interfaceneural network controlling conversion between a feature vector to beoutput from the first neural network and a feature vector to be input tothe second neural network; and transmit the input signal to the secondneural network.
 20. A signal processing apparatus comprising: at leastone processor configured to: receive a first signal; generate an inputsignal to be input to a target neural network by converting a firstfeature vector of the first signal to a second feature vector of theinput signal, based on parameters of the interface neural networkcontrolling conversion between different feature vectors; and transmitthe input signal to the target neural network.